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Abstract 

The purpose of this study was to examine individual differences in attitudes toward Audience 
Response Systems (ARSs) in secondary school classrooms. Specifically, the impact of gender, 
grade, subject area, computer comfort level, participation level, and type of use were examined in 
659 students. Males had significantly more positive attitudes toward ARSs than female students. 
Students who were more comfortable with computers had significantly more positive attitudes 
than students who were less comfortable. Students who did not actively participate in class before 
ARSs were used were more positive about this tool than students who regularly participated. 
Finally, students were significantly more positive about ARSs when they were used for formative 
(not for grades) as opposed to summative (for grades) assessment. There were no significant 
differences observed for grade level or subject area taught. 


Resume 

La presente etude visait a etudier les variations personnels dans la disposition des eleves du 
secondaire envers I’utilisation de systemes de reponse, ou televoteurs, en classe. Plus 
precisement, I’etude a examine I’impact du sexe, de I’annee d’etudes, de la matiere, de I’aisance 
en informatique, du niveau de participation et du type d’utilisation sur 659 eleves. La disposition 
des gargons envers les televoteurs etait significativement meilleure que celle des filles. De plus, 
les eleves les plus a I’aise avec les ordinateurs avaient une reponse significativement plus 
positive que les eleves moins a I’aise. Les eleves qui ne participaient pas activement en classe 
avant I’utilisation de televoteurs avaient quant a eux une attitude plus positive envers cet outil que 
les eleves qui participaient deja regulierement. Enfin, les eleves demontraient une attitude plus 
favorable envers les televoteurs lorsque ceux-ci etaient utilises aux fins devaluations formatives 
(non notees), par opposition aux fins devaluations sommatives (notees). II n’y avait pas de 
differences significatives observees en fonction de I’annee d’etudes ou de la matiere enseignee. 

Overview 

Audience Response System (ARSs), also known as clickers (Bergtrom, 2006) permit students to 
answer digitally-displayed multiple choice questions using a remote control. After students select 
their individual responses, the results are instantly presented in a bar chart that displays the 
number of students who selected each choice. The principal advantage of using an ARS is that it 
gives instant feedback to both students and instructors on how well the entire class understands 
concepts being presented using multiple choice questions. Once this feedback is attained, a 
teacher can alter the course of instruction or students can work out misconceptions via peer or 
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classroom discussion and further study. 

Researchers have concentrated on benefits, challenges, and strategies associated with ARS use 
(Caldwell, 2007; Fies & Marshall, 2006; Simpson & Oliver, 2007). Specific benefits have included 
increased attention during class (e.g., Siau, Sheng, & Nah, 2006; Slain, Abate, Hidges, 
Stamatakis, & Wolak., 2004), higher engagement in concepts presented (e.g., Bergtrom, 2006; 
Preszler, Dawe, Shuster, & Shuster, 2007; Simpson & Oliver, 2007), higher levels of participation 
(e.g., Draper & Brown, 2004; Greer & Heaney, 2004), increased attendance (e.g., Burnstein & 
Lederman, 2001; Greer & Heaney, 2004), improved quality of class discussion (e.g., Beatty, 2004; 
Brewer, 2004), and significantly higher learning performance on tests compared to traditional 
classes (e.g., Carnaghan & Webb, 2006; El-Rady, 2006; Latessa & Mouw, 2005). Challenges 
reported when an ARS is used have included technological “glitches” in the remote devices used 
to click in answers (e.g., Sharma, Khachan, Chan, & O'Byrne 2005; Siau, et al., 2006) and 
resistance to a new method of learning (e.g., Beatty, 2004). Finally, effective strategies used with 
ARSs have incorporated contingent teaching (e.g., Hinde & Hunt, 2006), formative assessment 
(e.g., Bergtrom, 2006; Dufresne & Gerace, 2004), and peer instruction (e.g., Beatty, 2004; Draper 
& Brown, 2004; Nicol & Boyle, 2003). 

One area, though, that has been largely overlooked is individual differences in attitudes toward 
the use of ARSs. Anecdotal evidence suggests that not all students respond in the same way 
when an ARS is used in the classroom. While attitudes toward this tool have been consistently 
positive, some students react negatively when ARSs are used for summative assessment (e.g., 
Caldwell. 2007), the technology does not work well (e.g., El-Rady, 2006; Hatch, Jensen, & Moore, 
2005), they have to adjust to a new style of teaching (e.g., Abrahamson, 2006), increased class 
discussion creates confusion (e.g., Nicol & Boyle, 2003, Reay, Bao, Li, Warnakulasooriya, & 
Baugh, 2005), and they have to put in the extra effort required to answer questions (e.g., Trees & 
Jackson, 2007). 

A systematic investigation of individual differences is important for at least three reasons. First, 
potential inequities can be identified. For example, males may like using ARSs more than 
females, a result that has not been tested, but one that has been observed with other technology- 
based educational tools (AAUW, 2000; Barker & Aspray, 2006; Kay, 2008; Whitley, 1997). If using 
ARSs in the classroom partially alienates female students, we need to re-consider how to use this 
tool or whether it is a viable option in the classroom. 

Second, understanding individual differences may help educators target and refine the use of 
ARSs in the classroom. It is unclear, for example, whether an ARS is a tool that can be used at all 
grade levels. It is conceivable that younger students may be too immature to handle the initial 
excitement created when an ARS is used. In addition, we do not know what subject areas might 
be best suited to the use of ARSs. The multiple choice questions used with an ARS may not be an 
ideal fit in all subject areas. 

Third, exploring the impact of individual differences in how an ARS is used is essential to refining 
and improving its use. No technology, including ARSs, inherently improves learning. The 
effectiveness of this technology in the classroom depends on the pedagogy employed (Caldwell, 
2007; Cuban, 2001; Russell, Bebell,, O’Dwyer, & O’Connor, 2003; Simpson & Oliver, 2007). 
Therefore it is important to compare the impact of different strategies used with ARSs. 

Literature Review 

Individual Differences in the Use of ARSs 

An extensive assessment of peer-reviewed articles on the use of ARSs in higher education (see 
Caldwell, 2007; Fies & Marshall, 2006; Judson & Sawada, 2002; Simpson & Oliver, 2007 for 
detailed reviews) coupled with an analysis of individual differences observed in computer-related 
behavior on other technologies has revealed at least six potential factors that could influence 
attitudes toward ARSs including gender, grade, subject areas, computer comfort, participation 
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level, and type of use. 


Gender 

Numerous studies have investigated the role of gender in computer behaviour (see AAUW, 2000; 
Barker & Aspray, 2006; Kay, 2008; Whitley, 1997 for detailed reviews). Most studies have looked 
at computer attitude, ability, and/or use. Roughly 30 to 50% of the studies report differences in 
favour of males, 10-15% in favour of females, and 40 to 60% no difference. Differences reported, 
while statistically significant, are often small. Overall, one could say there is a consistent pattern of 
small differences in computer attitude, ability, and use that favours males, however considerable 
variability exists. To date, no research has been done investigating gender differences in attitude 
toward ARSs. However, since most students have found this tool easy to learn and use (e.g., 
Hinde & Hunt, 2006; Pelton & Pelton, 2006; Sharma et al., 2005), it is speculated that gender will 
not play a significant role in the use of ARSs. 


Grade 

Regarding the impact of grade level or age on general computer attitudes, there has been limited 
research. Several studies reported that older students (15-16 years old) viewed computers as 
tools for accomplishing tasks and getting work done (e.g., word processing, programming, use of 
the Internet, and email), whereas younger students (11-12 years old) saw computers as a source 
of enjoyment (e.g., play games and use graphics software) (e.g., Colley, 2003; Colley & Comber, 
2003). 

With respect to ARSs, only one study has been done examining the role of grade level. Preszler 
et al. (2007) noted that third and fourth year university students were more resistant to and less 
accepting of ARSs than first and second year students. The older students also felt that ARSs 
were a distraction and had not helped them learn as much as the younger students felt they did. 
The authors speculated that older students may be more accustomed to a lecture mode of 
teaching and therefore may be more resistant to change than their younger counterparts. It is 
unclear how grade level would affect the use of ARSs in secondary school classrooms. 


Subject area taught 

Regarding research on general computer-related behaviors, no studies could be found looking at 
individual differences in computer behavior as a result of subject taught. ARS research is equally 
silent on individual differences in student attitudes as result of subject area. However, the range of 
subjects that have been used with ARSs in higher education is impressive and includes 
accounting, anatomy, biology, business, earth science, economics, environmental science, law, 
math, medicine, mathematics, pharmacology, philosophy, physics, physiology, and programming 
(Caldwell, 2007; Fies & Marshall, 2006; Simpson & Oliver, 2007). Because of the diversity of 
subjects where ARS has been used, it is worthwhile investigating the impact of subject area 
taught on student attitudes toward ARSs. 


Computer comfort 

Perceived comfort level with using computers has been shown to be particularly influential on 
computer ability and use (e.g., Barbeite & Weiss, 2004; Durndell & Haag, 2002; Shapka & Ferrari, 
2003; Solvberg, 2002). Research on computer comfort level and ARSs has not been done, 
however, many studies have reported that students find ARSs easy to learn and use (Hinde & 
Hunt, 2006; Pelton & Pelton, 2006; Sharma, et al., 2005). Therefore, even though computer 
comfort level has been a prominent factor in influencing computer-related behaviours in more 
advanced computer technologies, it is speculated that it will not play a major role in influencing 
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attitudes towards simple, easy to use ARS equipment. 


Participation level 

While considerable evidence suggests that higher education students participate more when 
ARSs are used in the classroom (Caldwell, 2007; Uhari, Renko, & Soini, 2003; Van Dijk, Van Den 
Berg, & Van Keulen, 2001), it is not clear how this research translates into a secondary school 
environment where classes are much smaller. For example, it is possible that students who 
habitually participate less may benefit more when using an ARS because responding is easy to do 
and anonymous. One of the goals of using ARSs is to increase student engagement, so it is 
prudent to investigate whether previous participation levels influence acceptance. 


Type of use 

A number of researchers have argued that the pedagogical strategies selected to integrate ARSs 
in the classroom have a fundamental influence on acceptance and overall success (Reay et al., 
2005; Simpson & Oliver, 2007; Stuart, Brown, & Draper, 2004). Caldwell (2007) noted that higher 
education students did not like using ARSs for participation grades or summative assessment 
(Caldwell. 2007), but that they reacted well to formative assessment (e.g., Bergtrom, 2006; 
Dufresne & Gerace, 2004). Furthermore, Tree and Jackson (2007) added that learning 
preferences had an impact on attitudes toward ARSs. Students were more positive about ARSs 
when they valued feedback, engagement and interaction. For the most part, though, limited 
research has been done examining and comparing specific strategies that affect attitudes toward 
ARSs. 

Summary and Purpose of Study 

The use of ARSs is a relatively new phenomenon, particularly in secondary schools. Previous 
research in higher education has focussed on the benefits, challenges and strategies with respect 
to using ARSs. Almost no research has been done examining individual differences in attitude 
toward ARSs. A review of the research on ARSs and computer technology in general revealed six 
promising factors that might account for variability in attitudes toward ARSs. 

The purpose of the current study was to investigate individual differences in secondary school 
students’ attitudes toward ARSs by focussing on gender, grade level, subject areas, computer 
comfort level, participation level, and type of use. 

Method 

Sample 

Students 

The student sample consisted of 659 students: 327 were males and 327 females. Five students 
chose not to enter their gender. Students were enrolled in grades 9 (n= 71), 10 (n=233), 11 
(n=149), and 12 (n= 206). ARSs were used in five different subject areas: business (n=66), 
computer technology (n=42), social science (n= 1 60), science (n=213), and math (n=134). Forty- 
four students did not identify a subject area. This missing subject data was dispersed across 
multiple classrooms. Eight-seven percent (n=572) of the students reported being comfortable or 
very comfortable with technology. Fifty-five percent of the students (n=364) reported that in a 
typical, non ARSs class, they participated a little or not at all. Data from the sample population 
was collected from 23 different classrooms in 15 different secondary schools. Teachers from 
these classrooms volunteered to participate in the study, and students were given the option of 
filling in a survey about ARS use. All students had to obtain signed parental permission to 
participate. 
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Teachers 

Twenty-three teachers (16 males, 7 females), with 1 to 26 years of teaching experience (M = 15.9, 
SD = 7.9), volunteered to participate in the ARS study. Most teachers reported that they were 
comfortable or very comfortable with technology (n=22, 96%). 


Procedure 

An educational coordinator emailed eligible secondary school teachers within an entire district and 
informed them of the ARSs study. All teachers were told that participation was voluntary and that 
they could withdraw from the study at any time. Participants received a half day of training in 
November on how to use the ARS software and a half day of training February on advanced use 
of ARSs in the classroom. They were then asked to use ARSs in their classrooms for a period of 
three months, although how often ARSs were used was dictated by each teacher. Each teacher 
shared a laptop computer, an LCD projector, and a one ARSs system with a partner. All students 
participated in ARS lessons, however, only those students with signed parental permission forms 
were permitted to fill in an anonymous, online survey about their use of the ARSs Note that when 
students were using ARSs, their answers were always anonymous to other students. 

The survey-based evaluation of ARSs was completed in the third month of use. During this month, 
94% (n=617) of secondary students reported using ARSs one to two times. Six percent (n=41) of 
the students used ARSs once a week. Note that each student filled out only one survey. 


Data Sources 
Student survey 

After using ARSs for three months, students completed the ARS Attitude Survey for Students (see 
Appendix A) targeting their last month of use. This survey consisted of nine, seven-point Likert 
scale items. Items were constructed based on a review of the ARS literature (Caldwell, 2007; Fies 
& Marshall, 2006; Simpson & Oliver, 2007) and focussed on general attitudes, student 
involvement, assessment, and learning. Internal consistency for the total nine-item scale was 0.89 
(Cronbach’s Alpha). 


Data Analysis 
Explanatory variables 

Six explanatory variables were examined in this study: gender (male, female), grade level (9, 10, 
11, and 12), subject area taught (business, computer technology, social science, math, and 
science), computer comfort level (item C in Appendix A), previous participation in non ARSs 
classes (low vs. high - item D in Appendix A) and type of use (formative assessment, formative 
and summative assessment, summative assessment). Low participation included “not at all” and 
“some of the time” responses from item D in Appendix A. High participation included “most of the 
time” and “all of the time” responses from item D in Appendix A. Type of use was determined by 
asking teachers whether they used ARSs for formal, graded tests (summative assessment) or to 
assess understanding without grades (formative assessment). 


Dependent variables 

The main dependent variable used in this study was total attitude score which was calculated by 
adding all nine items from the ARSs scale in Appendix A. A probability level of p < .001 was used 
to compensate for the number of independent t-tests or ANOVAs completed (see Kirk, 1982, p. 
102). If a significant difference was found within a specific explanatory variable, a formative 
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analysis of individual items from the ARS scale was completed using a MANOVA. Scheffe’s post- 
hoc analysis was used to assess differences among items in the ARS scale because it is one of 
the most flexible, conservative, and robust procedures (Kirk, 1982). In order to assess the relative 
importance of the six explanatory variables in predicting total attitude toward ARSs, a stepwise, 
linear regression analysis was done. 

Key Research Questions 

To examine individual differences in student attitudes toward the use of ARSs in secondary school 
classrooms, the following questions were addressed: 

1 . Are there differences in attitudes toward ARSs based on: 

o Gender? 
o Grade Level? 
o Subject Area Taught? 
o Computer Comfort Level? 
o Participation Level? 
o Type of Use? 

2. What are the relative contributions of gender, grade level, subject area taught, computer 
comfort level, participation level, and type of use in predicting attitudes toward ARSs? 

Results 

Overall Variability 

Before looking at specific sources that might contribute to explaining variability in attitudes toward 
ARSs, it is worth examining the overall variability of total attitude score toward ARSs. Scores 
ranged from a minimum of 9 to a maximum of 63 (possible range was 10 to 63) with a mean score 
of 44.4 (S.D. 10.1). Average item scores in the nine item scale ranged from 4.4 to 5.4 indicating 
that, overall, students were slightly positive about the use of ARSs (see Table 1). Because 
attitudes toward ARSs varied considerably, there is a need to explain observed variability. 

Table 1. Summary of Audience Response System Survey Questions 
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Summary of Audience Response System Survey Questions 


Question 

n 

Disagree 

Agree 

M 

S.D. 

Overall Attitude 

1. When clickers were used, the class was better. 

659 

16% 

52% 

4.7 

1.5 

Student Involvement 

2. 1 was more engaged in the lesson when clickers 

657 

10% 

75% 

5.3 

1.5 

were used. 

3. 1 was more motivated when clickers were used. 

656 

11% 

70% 

5.2 

1.5 

4. 1 participated more than I normally would when 

659 

15% 

55% 

5.1 

1.6 

clickers were used. 

5. Using clickers generated more class discussion. 

657 

18% 

55% 

4.6 

1.5 

Assessment 

6. Using clickers was a good way to test my 

655 

9% 

77% 

5.4 

1.4 

knowledge. 

7. 1 liked seeing what other students in the class 

656 

14% 

57% 

4.8 

1.5 

selected for answers. 

8. 1 liked using clickers for tests. 

636 

21% 

54% 

4.7 

1.8 

Learning 

9. 1 learned more when clickers were used. 

656 

23% 

44% 

4.4 

1.5 


Gender Differences 

With respect to total ARS attitude score, males (M= 46.1, S.D. = 9.5) had significantly more 
positive attitudes towards ARSs than females (M= 42.5, S.D. = 10.4) (t = 44.93, df = 615, p < 
001). The effect size of 0.36 is considered to be in the medium range by Cohen (1998). Since 
overall attitudes toward ARSs were different, a MANOVA was run to compare male and female 
students on the nine Likert-scale survey items examining attitudes toward using ARSs (Appendix 
A). Since Hotelling’s T was significant (p <.001 ), individual comparisons were done on each 
survey question. Males and females differed significantly on all but one of the items from the 
ARSs attitude survey (Table 2). Specifically, males were more involved when ARSs were used, 
liked using ARSs to test their knowledge, especially in summative evaluation, felt ARSs helped 
improve their learning, and overall, thought ARS-based classes were better than traditional 
classes. 

Table 2. ' Gender Differences in Attitudes toward Audience Response Systems 
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Gender Differences in Attitudes toward Audience Response Systems 



Females 


Males 



Measure 

M 

SD 

M 

SD 

F 

Overall Attitude 

1. When clickers were used, the 
class was better 

4.44 

1.51 

5.01 

1.43 

23.07 * 

Student Involvement 






2. I was more engaged in the 
lesson when clickers were 
used 

5.18 

1.56 

5.50 

1.35 

7.21 ** 

3. I was more motivated when 
clickers were used 

5.01 

1.56 

5.46 

1.43 

14.18 * 

4. I participated more than I 
normally would when clickers 
were used. 

4.97 

1.60 

5.29 

1.51 

6.53 *** 

5. Using the clickers generated 
more class discussion 

4.36 

1.48 

4.82 

1.71 

14.95 * 

Assessment 






6. Using the clickers was a good 
way to test my knowledge 

5.30 

1.49 

5.55 

1.33 

4.72 *** 

7. I liked seeing what other 
students in the class selected 
for answers. 

4.72 

1.49 

4.95 

1.52 

3.62 

8. I liked using clickers for tests 

4.41 

1.86 

5.13 

1.73 

24.68 * 

Learning 






9. I learned more when clickers 
were used 

4.17 

1.51 

4.60 

1.44 

13.11 * 


*p < .001; “ p < .005; *** p < .05 


1 This table is a modified version of Table 5 from Kay, R. H. (in press). | 

Grade Level Differences 

Regarding total ARS attitude score, there were no significant differences among grade 9 (M= 
46.1, S.D. = 9.5), grade 10 (M= 46.0, S.D. = 8.7), grade 11 (M= 48.2, S.D. = 9.6), and grade 12 
(M= 46.3, S.D. = 9.3) students, therefore individual scale items were not analyzed. 


Subject Area Differences 

A one way ANOVA for total ARS attitude score showed significant differences among subject area 
(F = 10.29, p < .001). Scheffe’s post hoc analysis revealed that students using ARSs in social 
science (M= 47.8, S.D. = 8.6) had significantly more positive attitudes than students using ARSs 
in either science (M= 42.8, S.D. = 10.6) or mathematics (M= 41.2, S.D. = 10.5) with effect sizes 
(0.52 to 0.69 respectively) in the medium range according to Cohen (1998). In addition, students 
of computer technology (M= 47.6, S.D. = 6.9) had significantly more positive attitudes toward 
ARSs than students of mathematics (M= 41.2, S.D. = 10.5). The effect size of 0.72 was also 
considered to be in the medium range (Cohen, 1998). 
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A MANOVA was run to compare subject area taught on the nine Likert-scale survey items 
examining attitudes toward using ARSs (Appendix A). Hotelling’s T was significant (p < .001), so 
individual comparisons were done on each survey question. Social science student attitudes 
toward ARSs were consistently higher than mathematics and science students on most survey 
items (see Table 3). 

Table 3. MANOVA Examining Attitude toward Audience Response Systems as a Function 
of Subject Area Taught 

MANOVA Examining Attitude toward Audience Response Systems as a Function of Subject Area 
Taught 


Source 

df 

SS 

F 

Scheffe’.s Post Hoc Analysis (p <.05) 

Overall Attitude 





1 . When clickers were used, 
class was better. 

4 

11.6 

5.2* 

Soc Sci. > Math & Science 

Student Involvement 





2. 1 was more engaged in the 
lgggjafl when clickers were 
used. 

4 

10.0 

4.7** 

Soc Sci. > Math & Science 

3. 1 was more motivated 
when clickers were used. 

4 

14.6 

6.6* 

Soc Sci. > Math & Science 

4. 1 participated more than I 
normally would when 
Clickers were used. 

4 

13.7 

y» 

bo 

* 

Soc Sci. > Math & Business 

5. Using clickers generated 
inSJS class discussion. 

4 

11.8 

5.6* 

Soc Sci. & Computers > Math 

Assessment 





6. Using clickers was a good 
jygy. to test my knowledge. 

4 

11.5 

6.0* 

Soc Sci. > Math & Science 

7. 1 liked seeing what other 
students in the class 
SSlSStSd for answers. 

4 

6.8 

3.1 *** 

Soc Sci. > Math 

8. 1 liked using clickers for 
!£§!&.• 

4 

42.2 

13.8 * 

Soc Sci. & Computer & Math > Science 

Learning 





9. 1 learned more when 
fiUgtea&uscd. 

4 

17.6 

8.1 * 

Soc Sci. & Computer & Business > Math 


*p < . 001 ; "p < . 005 ; *"p <.05 


Computer Comfort 

An independent samples t-test revealed that students who were comfortable with technology (M= 
45.4, S.D. = 9.5) were significantly more positive about ARSs than students who were not 
comfortable with technology (M= 37.9, S.D. = 11.3; t=-6.46, p < .001)). The effect size of 0.72 is 
considered in the medium range (Cohen, 1988). 
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A MANOVA, used to compare students who were not comfortable with technology (n=80) with 
students who were comfortable with technology (n=534) on the nine Likert-scale survey items 
examining attitudes toward using ARSs (Appendix A), produced a Hotelling’s T that was 
significant (p < .001). Therefore, individual comparisons were examined for each item. Students 
who were more comfortable with technology were more involved when ARS was used, liked using 
ARSs to test their knowledge, particularly with summative evaluation, felt ARSs helped improve 
their learning, and overall, believed that ARS-based classes were better than traditional classes. 

Table 4. 2 Differences on in Attitudes toward Audience Response Systems based on 
Computer Comfort Level 

Differences on in Attitudes toward Audience Response Systems based on Computer Comfort Level 



Not Comfortable 

Comfortable 


Survey Item 

M 

SD 

M 

SD 

F 

Overall Attitudes 






1 . When clickers were used, the 
class was better. 

4.13 

(1.46) 

4.82 

(1.49) 

15.39* 

Student Involvement 






2. 1 was more engaged in the 
lesson when clickers were used. 

4.51 

(1.79) 

5.47 

(1.36) 

31.63 * 

3. 1 was more motivated when 
clickers were used. 

4.28 

(1.82) 

5.38 

(1.40) 

39.86 * 

4. 1 participated more than I 
normally would when clickers 
were used. 

4.36 

(1.82) 

5.25 

(1.49) 

23.42 * 

5. Using clickers generated more 
class discussion. 

4.05 

(1.48) 

4.68 

(1.47) 

12.54* 

Assessment 






6. Using clickers was a good way 
to test my knowledge. 

4.71 

(1.71) 

5.54 

(1.34) 

24.65 * 

7. 1 liked seeing what other 
students in the class selected 
for answers. 

4.60 

(1.33) 

4.89 

(1.51) 

2.59 

8. 1 liked using clickers for tests. 

3.78 

(2.01) 

4.92 

(1.76) 

28.53 * 

Learning 






9. 1 learned more when clickers 
were used. 

3.80 

(1.59) 

4.47 

(1.47) 

14.36 * 


* p < .001 

2 This table is a modified version of Table 6 from Kay, R. H. (in press). | 


Participation Level 
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In this study, students were asked their level of participation in a typical class before the study 
started. Three-hundred and forty-eight students reported that they never or occasionally 
participated in class (Low Participation), whereas 273 students said they almost always or always 
participated (High Participation). The independent samples t-test revealed low participation 
students (M= 45.1, S.D. = 10.1) were significantly more positive about ARSs than high 
participation students (M= 43.6, S.D. = 10.1, t=2.02, p < .05). The effect size of 0.16 is considered 
small according to Cohen (1988). 

The participation level differences were not significant to the prescribed p < .001 level, however, 
because this study is a first attempt to examine individual differences, a MANOVA was run to 
compare low participation versus high participation students on the each of the nine, Likert-scale 
survey items examining attitudes toward using ARSs (Appendix A). Hotelling’s T was significant (p 
<. 001), so individual comparisons were examined. Low participation students felt they 
participated more (p < .001), liked using ARSs more for tests (p < .05), and thought they learned 
more when ARSs were used (p < .05). All other items showed no significant difference between 
low and high participation students. 


Type of Use 

A one way ANOVA for total ARS attitude score showed significant differences among type of use 
(F = 40.14, p < .001). Scheffe’s post hoc analysis revealed that students were significantly more 
positive about ARSs when formative assessment was used (M= 46.7, S.D. = 9.3), as opposed to 
mixed assessment (M= 42.3, S.D. = 9.4, p < .05, Cohen’s d = 0.47) or summative assessment 
(M= 37.7, S.D. = 10.6, p < .05, Cohen’s d = 0.90). In addition, students who experienced the 
mixed assessment reported significantly more positive attitudes than students who experienced 
summative assessment only (p < .05; Cohen’s d = 0.46). Means for all ARSs Likert item scales 
based on strategy selected are presented in Table 5. Note that the means for most items show a 
steady rise in value from summative, to mixed, to formative assessment. 

Table 5. 3 Mean Survey Item Scores as a Function of Strategy Used with Audience 
Response Systems 
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Mean Survey Item Scores as a Function of Strategy Used with Audience Response Systems 


Survey Item 

Summative 

Assessment 

Mixed 

(Formative & 
Summative) 

Formative 

Assessment 


M 

SD 

M 

SD 

M 

SD 

Overall Attitudes 







1. When clickers were used, the 
gjgsjjs, was better. 

4.19 

(1.46) 

4.43 

(1.32) 

4.97 

(1.50) 

Student Involvement 







2. 1 was more engaged in the 
lgggQjj, when clickers were used. 

4.52 

(1.57) 

5.05 

(1.44) 

5.65 

(1.33) 

3. 1 was more motivated when 
SUSJSSJS were used. 

4.44 

(1.60) 

4.79 

(1.47) 

5.57 

(1.39) 

4. 1 participated more than I 
normally would when clickers 
vvgjgused. 

4.39 

(1.51) 

4.85 

(1.56) 

5.41 

(1.50) 

5. Using the clickers generated 
more class discussion. 

3.83 

(1.35) 

4.45 

(1.37) 

4.85 

(1.49) 

Assessment 







6. Using clickers was a good way 
tp test my knowledge. 

4.56 

(1.59) 

5.27 

(1.39) 

5.71 

(1.28) 

7. 1 liked seeing what other 
students in the class selected 
fet answers. 

4.10 

(1.47) 

5.23 

(1.43) 

4.92 

(1.48) 

8. 1 liked using clickers for tests. 

4.09 

(2.16) 

4.19 

(1.93) 

5.11 

(1.62) 

Learning 







9. 1 learned more when clickers 

Soused' 

3.63 

(1.48) 

4.13 

(1.37) 

4.65 

(1.46) 


3 This table is a modified version of Table 7 from Kay, R. H. (in press). | 

A MANOVA was run to compare these three approaches to using ARSs on the nine Likert scale 
questions assessing attitude toward ARSs (Table 6). Using ARSs for formative assessment was 
rated significantly more positively than summative assessment for all items on the ARSs attitude 
scale. In addition, a formative assessment approach showed significantly higher ratings than a 
mixed method (formative and summative) approach on most attitude scale items. 

Table 6. MANOVA Examining Attitude toward Audience Response Systems as a Function 
of Teaching Strategy 
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MANOVA Examining Attitude toward Audience Response Systems as a Function of Teaching 
Strategy 


Source 

df 

SS 

F 

Scheffe’s Post Hoc Analysis (p <.05) 

Overall Attitude 





1. When ICCS was used, 
foe class was better. 

2 

62.0 

14.6* 

Formative > Mixed & Summative 

Student Involvement 





2. 1 was more engaged in the 
when clickers were 

used. 

2 

115.2 

29.6* 

Formative > Mixed > Summative 

3. 1 was more motivated 
when clickers were used. 

2 

132.1 

31.9* 

Formative > Mixed & Summative 

4. 1 participated more than I 
normally would when 
SlieKferg. were used. 

2 

97.7 

21.3 * 

Formative > Mixed > Summative 

5. Using clickers generated 
more class discussion. 

2 

89.2 

21.4* 

Formative & Mixed > Summative 

Assessment 





6. Using clickers was a good 
Sax. to test my knowledge. 

2 

111.0 

30.3 * 

Formative > Mixed > Summative 

7. 1 liked seeing what other 
students in the class 
SSfeSlSd for answers. 

2 

76.0 

17.6* 

Formative & Mixed > Summative 

8. 1 liked using clickers for 

tests- 

2 

130.4 

20.6* 

Formative > Mixed & Summative 

Learning 

9. 1 learned more when 
elicits were used. 

2 

94.5 

22.6* 

Formative > Mixed > Summative 


* p < .001 


Predicting Total Attitudes toward ARSs 

A stepwise multiple regression analysis was done to examine the relationship among the six 
individual difference factors examined in this paper (gender, grade level, subject area, computer 
comfort level, participation level, type of use ) and total attitude score on the ARS scale (Appendix 
A) The stepwise method was chosen because there was no previous theory to guide selection of 
predictors Four predictors were identified as being significant in the regression equation including 
type of use, computer comfort, participation and gender. Grade level and subject area were not 
significant predictors (see Table 7). Note that the total variance accounted for by the final 
regression equation was small at only 20%. 

Multicollinearity did not appear to be a problem because no VIF (variance inflation factor) was 
over the acceptable level of 10 (Bowerman & O’Connell, 1990) and the average VIF (M=1.0) was 
not greater than one (Bowerman & O’Connell, 1990) when all variables were entered into the 
equation. The Durbin-Watson test produced a value of 1.52 indicating no problems with respect to 
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autocorrelation of errors (Durbin & Watson, 1951). 

Table 7. Stepwise Regression Equation Predicting Total ICCS Attitude Score 

Stepwise Regression Equation Predicting Total ICCS Attitude Score 


Predictor 

R 

R 2 Change 

B Weight 

SE a 

SRC b 

t 

Type of Use 

0.36 

0.128 

-4.28 

0.50 

-0.33 

-8.62 * 

Comp. Comfort 

0.42 

0.046 

6.43 

1.16 

0.21 

5.57* 

Participation 

0.43 

0.014 

-2.56 

0.78 

-0.13 

-3.29 ** 

Gender 

0.45 

0.011 

-2.16 

0.78 

-0.11 

-2.77 *** 


a SE - Standard Error 


b SRC - Standard Regression Coefficient 
* p < .001; * * p < .005; *** p < .01 


Discussion 

The purpose of this study was to examine individual differences in attitudes toward ARSs. Six 
explanatory variables were chosen: gender, grade level, subject area, computer comfort level, 
participation level before the study, and type of use. The impact of each of these variables will be 
discussed. 


Gender 

Gender differences in computer attitudes over the past 25 years have been small but persistent in 
favour of males (AAUW, 2000; Barker & Aspray, 2006; Kay, 2008; Whitley, 1997). Differences in 
attitudes toward ARSs between males and females are consistent with previous research. Overall, 
males were significantly more positive than females toward ARSs, however, the effect size was 
moderate. In addition, males were significantly more positive on all but one item from the ARS 
attitude scale. 

Because previous gender differences in computer attitudes is often associated with confidence 
and self-efficacy (AAUW, 2000; Barker & Aspray, 2006; Kay, 2008; Whitley, 1997), it is somewhat 
surprising that males were more positive than females about ARSs given that this technology has 
been reported by students to be easy to learn and use (e.g., Hinde & Hunt, 2006; Pelton & Pelton, 
2006; Sharma, et al., 2005; Siau, et al., 2006). However, the regression analysis done with all six 
variables indicated type of use, computer comfort, and previous participation levels may be more 
important than gender in terms of influencing attitude toward ARSs. It is important to gather more 
qualitative evidence, perhaps in the form of interviews or focus groups, to identify why females 
may be more uncomfortable with ARSs. 


Grade Level 

Research on the impact of age or grade level on attitudes toward computers and ARSs has 
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presented a mixed picture. On the one hand, younger students have been observed to be more 
enthusiastic about computers (e.g., Colley, 2003; Colley & Comber, 2003). On the other hand, 
Preszler et al. (2007) suggest that older students might be more resistant to ARSs because they 
are accustomed to other methods of teaching. Results from the current study, indicate that grade 
had no impact on attitudes towards ARSs in a secondary school environment. It is possible that 
secondary school students are more accepting of new technology, regardless of age. Moreover, 
they may have had less time than higher education students to settle into preferred learning 
patterns or teaching styles. 


Subject Area Taught 

Little research has been done to date examining attitude toward ARSs among different subject 
areas. In this study, secondary students in social sciences classes were consistently more 
positive about ARSs than either mathematics or science students. However, the regression 
analysis comparing all six variables indicated that subject area was not significant predictor of 
attitude toward ARSs. It is possible that type of use was an overriding factor. A quick look at the 
data reveals that in social science courses, summative assessment was used only 17% of the 
time, whereas in mathematics and science courses it was used over 35% of the time. Therefore, 
the main reason that social science students have more positive attitude toward ARSs might 
reflect how ARSs were used rather than the subject area taught. More systematic research is 
needed to determine if certain subject areas are more naturally disposed to the use of multiple 
choice questions for guiding instruction. 


Computer Comfort 

Secondary students who were more comfortable with computers were more positive about ARSs 
on eight of the nine scale items in Appendix A. This result is consistent with previous research on 
computer comfort level and computer related behaviour (e.g., Barbeite & Weiss, 2004; Durndell & 
Haag, 2002; Shapka & Ferrari, 2003; Solvberg, 2002), although somewhat unexpected given that 
ARSs are easy to use and learn. It is important to note, though, that only 13% of the total sample 
was uncomfortable with computers, so negative attitudes are not the norm. More research is 
needed in the secondary school environment to explore why differences exist so that educators 
can help accommodate less technologically savvy students who may be disadvantaged when 
using ARSs. 


Participation Level 

Students who normally participate less in a traditional secondary school classroom environments 
appear to benefit more from ARSs in certain keys areas. They participate more in class, they like 
using ARSs more for tests, and they feel that they learn more than students who participate more 
than they do in a traditional class. This result makes some sense because responding to ARS 
questions is easy and anonymous. Students, who are normally reluctant to participate for fear of 
being embarrassed by an incorrect answer in front of the class, do not have to worry about 
exposure when an ARS was used. Furthermore, increased participation using the remote devices 
may have convinced “low participation” students that they were learning more. The impact of 
previous participation level, while somewhat narrow, was strong enough to be included in the final 
regression equation as a significant predictor of attitude toward ARSs. 


Type of Use 

While pedagogy is considered critical when using ARSs (Reay et al., 2005; Simpson & Oliver, 
2007; Stuart et al., 2004), limited research has been done comparing different teaching strategies. 
The data in this study suggest that students prefer formative assessment over summative 
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assessment when using ARSs. In addition, type of use appears to be one of the principle 
predictors of attitudes when all variables are entered into a regression equation. Simply using an 
ARS does not guarantee positive attitudes - the type of strategy employed influences secondary 
school students’ overall perception of effectiveness. While it may seem like a time saving 
approach in terms of grading to use ARSs for formal tests, many secondary school students 
appear not to like this practice. 


Comparing All Six Sources of Individual Difference in Attitudes toward ARSs 

The regression analysis indicated that type of use, computer comfort, previous participation level, 
and gender were significant predictors of attitudes toward ARSs, whereas grade level and subject 
area were not. This result is consistent with the individual analyses of each variable. It is difficult, 
though, to accurately estimate the relative importance of the predictor variables. It is noteworthy 
that all four predictors accounted for only 20% of the total variance in attitudes toward ARS. More 
research is needed to identify other factors that might influence the acceptance of ARSs. 


Caveats and Future Research 

This study was a first attempt to investigate individual differences in attitudes toward ARSs in 
secondary school classrooms. Analysis of the data was based on a comprehensive review of the 
literature examining the use of ARSs in higher education. There are a number of caveats, though, 
that need to be considered when interpreting the results. 

First, the results are based on one data collection source- survey questions. While the survey 
was judged to be reliable, triangulation of data collection tools is recommended for future studies. 
Specifically, qualitative data is needed to understand some key questions. For example it is 
unclear why gender differences and computer comfort level influence student attitudes when 
using an ARS is so easy to learn and use. 

Second, the results are based on limited use of ARSs - one or two times in a month. The impact 
on attitudes could be decidedly different when ARSs are used on a regular basis. For example, 
long term use might attenuate the impact of gender and computer comfort level once students 
became used to using ARSs. 

Third, a wider variety of teaching strategies used with ARSs should be examined in order to 
acquire a more complete picture of effectiveness and individual differences. One promising 
strategy that has yet to be examined in secondary school, but which has received considerable 
acclaim in higher education, is peer-based teaching. Peer-based instruction involves displaying a 
higher-level question that could identify a misconception, asking students to click in an answer, 
giving students time to discuss and defend their answers with two to four peers, taking a re-vote 
on the original question, and having the instructor provide a brief summary. A number of 
researchers have reported considerable success with this approach (Draper & Brown, 2004; 
Hinde & Hunt, 2006; Pelton & Pelton, 2006; Webking & Valenzuela, 2006). 

Fourth, a more detailed, survey of ARSs attitudes needs to be developed where the construct 
validity of specific factors such student involvement, assessment and learning is tested. The 
measure of total attitude used in this study was considered reliable, but the validity of individual 
items assessed was not established. 

Finally, a good next step in researching individual differences associated with the use of ARSs is 
to look at actual learning behaviour and performance. While there may be an implicit assumption 
that individual differences in attitudes affect overall learning experience, this assumption has yet 
to be tested. It is not clear whether individual differences in attitudes based on type of use, 
computer comfort level, previous participations rates, and gender translate to changes in actual 
learning. 
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Summary 

The purpose of this study was to investigate individual differences in attitudes toward ARSs in 
secondary school classrooms. Six explanatory variables were looked at including gender, grade 
level, subject area, computer comfort, previous participation levels, and type of use. The results 
indicated that males, students with a high computer comfort level, and students who do not 
actively participate in class had significantly higher positive attitude toward ARSs. It was also 
observed that students were significantly more positive about using ARSs when it was used for 
formative as opposed to summative assessment. There were no significant differences observed 
for grade level or subject area taught. 
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Appendix A - ICCS Attitude Survey for Students 

A. What grade are you in? 

B. Gender (circle one) Male Female 

C. How comfortable are you with technology? (circle one) 

Not at all comfortable Somewhat Comfortable Comfortable Very Comfortable 

E. How often did you use ICCS in the past month? 

Never 1-2 times Once a week 2-3 times per week 


Item 

Strongly 

Disagree 

Disagree 

Slightly 

Disagree 

Neutral 

Slightly 

Agree 

Agree 

Strongly 

Agree 

General Attitude 








1. When ICCS was used, the class 
was better 

1 

2 

3 

4 

5 

6 

7 

Student Involvement 








2. I was more engaged in the lesson 
when ICCS was used. 

1 

2 

3 

4 

5 

6 

7 

3. I was more motivated when ICCS 
was used. 

1 

2 

3 

4 

5 

6 

7 

4. I participated more than I 

normally would when ICCS was 
used. 

1 

2 

3 

4 

5 

6 

7 

5. Using ICCS generated more class 
discussion 

1 

2 

3 

4 

5 

6 

7 

Assessment 








6. Using ICCS was a good way to 
test my knowledge. 

1 

2 

3 

4 

5 

6 

7 

7. I liked seeing what other students 
in the class selected for answers. 

1 

2 

3 

4 

5 

6 

7 

8. I liked using ICCS for tests. 

1 

2 

3 

4 

5 

6 

7 

Learning 








9. I learned more when ICCS was 
used 

1 

2 

3 

4 

5 

6 

7 
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